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ABSTRACT 


Fast  Patrol  Boats  were  considered  a  negligible  threat  when  the  U.S.  Navy  focused 
on  blue  water  operations  away  from  shore.  Now  that  the  Navy’s  focus  has  shifted  to  the 
littorals,  where  these  ships  patrol,  the  Fast  Patrol  Boat’s  potential  as  a  credible  adversary  is 
gaining  acceptance.  Moreover,  the  threat  may  be  greatly  enhanced  if  Fast  Patrol  Boats 
employ  Commercial  Off-The-Shelf  Command  and  Control  equipment  to  coordinate  their 
efforts.  This  paper  presents  the  design  and  results  of  a  wargaming  experiment  conducted 
with  Batman  &  Robin  at  the  Naval  Postgraduate  School  to  examine  this  issue. 

The  research  question  is  how  would  U.S.  Navy  Surface  Action  Groups  perform 
against  Fast  Patrol  Boats  equipped  with  Commercial  Off-The-Shelf  Command  and  Control 
equipment  which  enables  these  comparatively  unsophisticated  ships  to  conduct  coordinated 
attacks.  The  experiment  also  looked  at  two  other  factors:  Surface  Action  Group  work  load, 
which  might  exacerbate  the  offensive  capability  of  Fast  Patrol  Boats  conducting 
coordinated  attacks  with  advanced  Command  and  Control;  and  the  quality  of  information 
our  Surface  Action  Groups  are  provided  which  could  likewise  play  a  pivotal  role  in  a 
confrontation  or  engagement. 

A  two-cubed  factorial  experiment  was  conducted  to  test  seven  hypotheses.  Data 
were  collected  on  ten  performance  measures  for  128  trials  total.  Significant  results  were 
obtained  for  three  factors  and  three  interactions.  Operational  explanations  are  provided. 
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EXECUTIVE  SUMMARY 


Fast  Patrol  Boats  (FPB)  have  always  been  an  effective  platform  for  littoral  warfare. 
With  today’s  advanced  commercial  off-the-shelf  command  and  control  (C^)  technology,  the 
threat  FPBs  pose  to  the  U.S.  Navy  could  drastically  increase.  Effective  would  enable 
FPBs  to  conduct  coordinated  attacks  using  sophisticated  tactics  that  could  saturate  a  Surface 
Action  Group’s  (SAG)  defenses. 

In  addition,  the  small  size  and  high  speed  of  FPBs  combine  to  complicate 
intelligence  gathering  and  increase  the  work  load  on  SAG  operators  trying  to  mflintain  an 
accurate  surface  picture  of  the  littoral  environment. 

The  main  research  question  is  the  effect  that  coordinated  attacks  by  FPBs, 
employing  advanced  C^,  will  have  in  an  engagement  with  a  U.  S.  Navy  SAG  operating  in 
littoral  seas.  Also  of  interest  are  the  effects,  and  their  interactions  with  the  main  effect,  of 
two  elements  which  accompany  the  presence  of  FPBs  and  a  littoral  operating  area: 
increased  work  load  and  incomplete  information. 

FPB  was  modeled  by  varying  the  tactical  coordination  of  the  FPBs  between  high 
and  low.  High  coordination  was  modeled  by  three  concurrent  waves  of  attacks  of  FPBs 
with  most  of  the  FPBs  in  a  restricted  emissions  status.  Low  coordination  was  modeled  by 
sequential  attacks  of  FPBs.  Work  load  was  modeled  by  varying  the  number  of  neutral  ships 
in  the  vicinity  of  the  attack.  High  work  load  scenarios  contained  24  neutral  ships  in  the 
operating  area  while  low  work  load  scenarios  contained  only  nine.  Information 
completeness  was  modeled  by  varying  the  completeness  of  the  maritime  intelligence  reports 
that  the  SAG  received  during  the  trial.  High  information  completeness  was  represented  by 
providing  the  subjects  with  a  ratio  of  hostile  ships  in  each  threat  sector  to  the  total  number 
of  ships  in  the  threat  sector.  Low  information  completeness  was  represented  by  only 
providing  the  total  number  of  ships  in  each  threat  sector. 

A  two-cubed  factorial  experiment  was  conducted  to  test  the  seven  hypotheses  (three 
main  hypotheses  plus  four  secondary  hypotheses  examining  all  possible  interactions).  Eight 
Persian  Gulf  scenarios  were  developed  on  the  wargame  simulator  Batman  &  Robin  as  a 


XV 


context  for  the  experiment.  The  wargame  has  two  modes  of  operation:  Batman,  where 
scenarios  are  executed,  and  Robin,  where  scenarios  are  developed.  Each  scenario 
represented  a  imique  combination  of  the  three  factors  at  two  levels.  Sixteen  subjects  from 
the  Naval  Postgraduate  School  (NPS)  participated  in  the  experiment.  Each  subject  played 
eaeh  scenario  using  Batman  once.  Trials  (8x16=  128)  were  run  at  four  times  real  time  and 
were  completed  in  38  minutes. 

In  each  scenario  the  Blue  force  SAG  was  comprised  of  three  combatants  and  five 
helicopters.  Red  forces  were  comprised  of  nine  FPBs.  The  subjects  controlled  the  Blue 
order  of  battle.  Red  order  of  battle  was  scripted  in  Robin  during  scenario  development  but 
could  deviate  from  the  script  in  Batman  according  to  its  pre-programmed  artificial 
intelligence. 

Ten  performance  measures  were  automatically  generated  by  Batman  &  Robin  at  the 
end  of  each  trial  and  were  used  to  test  the  hypotheses.  The  data  analysis  plan  includes  use 
of  frequency  plots,  box  plots,  MANOVA,  univariate  ANOVA,  interaction  plots,  residual 
plots,  and  non-paramtetric  statistics  to  assess  the  data.  A  significance  level  of  a  =  0.05  was 
established  to  test  all  null  hypotheses. 

Hypotheses  tests  show  significant  results  in  the  main  hypotheses  and  several 
interaction  hypotheses  over  a  broad  range  of  performance  measures.  The  most  important 
findings  are  condensed  as  follows: 

•  When  the  FPBs  had  high  C^,  fewer  were  detected,  however,  of  those  detected 
the  range  of  detection  was  greater. 

•  When  the  FPBs  had  high  C^,  fewer  were  destroyed,  but  those  that  were 
destroyed  were  done  so  at  a  greater  range. 

•  The  range  at  which  the  FPBs  were  detected  and  destroyed  both  decreased  when 
work  load  increased. 

•  With  complete  information  the  FPBs  were  destroyed  at  a  greater  range. 
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I.  INTRODUCTION 


This  chapter  discusses  the  purpose  and  scope  of  the  experiment  and  closes  with  the 
organization  of  the  rest  of  the  thesis. 

A.  PURPOSE 

This  section  lays  out  the  purpose  of  the  experiment  by  describing  the  problem  in 
real  world  terms.  This  leads  to  a  statement  of  the  research  question,  and  the  approach  taken 
to  examine  the  problem.  The  three  main  factors  and  their  levels  are  discussed  with 
anticipated  results  provided. 

1.  Real  World  Problem 

As  more  and  more  navies  turn  away  from  blue  water  naval  strategies  that 
necessitate  sophisticated  and  expensive  multipurpose  combatants  and  toward  regional  and 
littoral  dominance,  the  fast  patrol  boat’s  utility  as  a  cost  effective  naval  platform  for  littoral 
warfare  increases  [Ref.  1],  Coupled  with  the  availability  of  commercial  off-the-shelf 
(COTS)  command  and  control  (C^)  technology,  fast  patrol  boats  present  a  unique  challenge 
to  the  U.S.  Navy  and  the  traditional  maritime  powers  still  operating  large  naval  combatants 
built  for  blue  water  operations  [Ref.  2]. 

Moreover,  the  U.S.  Navy  frequently  deploys  Surface  Action  Groups  or  SAGs  (task 
groups  comprised  primarily  of  combatants)  throughout  the  littoral  regions  of  the  world. 
Despite  the  fact  that  SAGs  have  organic  helicopter  assets,  without  the  protection  of  an 
aircraft  carrier’s  air  wing  they  are  vulnerable  to  stealthy,  formidably  armed  fast  patrol  boats 
(FPB).' 

The  FPB’s  small  size  and  high  speed  combine  to  complicate  intelligence  gathering. 
In  addition,  U.S.  Naval  combatants  are  not  optimized  to  track,  target  and  neutralize  small, 
fast  surface  platforms,  nor  can  they  still  assume  the  dominant  tactical  advantage  inherent 

1  The  helicopter  asset  in  mention  is  the  LAMPS  MK  III  SH-60B  Seahawk.  Currently  the  Seahawk  has  no  lethal 
ASUW  capability.  The  SH-60B  Block  I  upgrade  will  have  a  forward  firing  weapon’s  capability.  It  is  scheduled 
for  fleet  introduction  by  1998  [Ref  3]. 


1 


with  superior  [Ref.  4].  Still  employing  forces  specifically  designed  for  blue  water 
warfare,  the  U.S.  Navy  is  caught  in  the  middle  of  this  shifting  paradigm  and  appears  slow  to 
react  to  the  FPB  threat  in  the  littorals  [Ref  5]. 

2.  Research  Question 

FPBs  already  possess  speed  and  stealth  advantages  over  most  combatants.  The 
main  research  question  concerns  the  effect  that  coordinated  saturation  attacks  by  FPBs, 
employing  advanced  C\  will  have  on  an  engagement  with  U.  S.  SAGs  operating  in  littoral 
seas.  Also  of  interest  are  the  effects  of  increased  workload  and  incomplete  intelligence, 
both  of  which  accompany  the  presence  of  FPBs  in  a  littoral  operating  area,  and  their 
interactions  with  the  main  effect. 

3.  Approach 

A  three  factor  experiment  was  constructed  involving  human  decision  makers.  All 
factors  were  presented  at  two  levels,  high  and  low,  resulting  in  a  two  cubed  factorial 
experiment.  The  experiment  was  designed  and  executed  on  the  Battle  Management 
Assessment  System  and  Raid  Originator  Bogie  Ingress  wargame,  more  commonly  called 
Batman  and  Robin  (B&R).  The  wargame  has  two  operational  interfaces.  In  Robin, 
scenarios  are  developed  and  edited.  In  Batman,  scenarios  are  executed  and  operational 
performance  is  measured.  Eight  different  scenarios  were  generated  in  Robin  to  present  all 
combinations  of  levels  of  factors.  Each  scenario  was  then  administered  to  16  Naval 
Postgraduate  School  (NPS)  officer  students  using  Batman,  for  a  total  of  128  trials.  The 
geographic  location  of  each  scenario  was  identical:  the  Persian  Gulf,  between  the  Straits  of 
Hormuz  and  Bahrain.  Red  forces  were  comprised  of  nine  OSA II  FPBs.  A  three  combatant 
SAG  composed  of  one  Kidd  class  destroyer,  two  Spruance  class  destroyers,  and  five  SH- 
60B  Seahawk  helicopters  constituted  Blue  forces.  All  forces  were  armed  and  equipped 
reeilistically.  The  three  design  variables  or  factors  were: 

a.  Information  Completeness 

For  high  information  completeness,  the  maritime  intelligence  was 
represented  by  a  ratio  of  hostile  ships  in  a  threat  sector  to  the  total  number  of  ships  in  the 
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threat  sector.  For  low  information  completeness,  only  the  total  number  of  ships  per  threat 
sector  was  given. 

b.  Workload 

High  workload  scenarios  were  characterized  by  24  neutral  ships  and  nine 
hostile  FPBs  in  the  operating  area.  Low  work  load  scenarios  had  nine  neutral  ships  and 
nine  hostile  FPBs  in  the  operating  area. 

c.  FPB  Cf 

The  third  factor  was  FPB  C^.  This  is  not  only  the  primary  factor  of  interest, 
but  also  the  original  impetus  for  the  experiment.  The  analysis  will  focus  on  this  factor. 
Two  levels  of  FPBs  were  achieved  by  varying  their  tactical  coordination  between  high 
and  low.  High  coordination  was  modeled  by  three  concurrent  waves  of  attacks  of  FPBs. 
For  this  case,  seven  of  the  nine  FPBs  were  in  restricted  emission  control  (EMCON)  status 
with  the  remaining  two  in  an  unrestricted  EMCON  status.  It  was  assumed  that  coordinated 
attackers  would  share  information  about  their  enemy,  the  U.S.  SAG  in  this  case,  and  thus 
require  fewer  emissions.  Low  coordination  was  modeled  by  sequential  attacks  of  FPBs 
with  only  two  of  the  nine  FPBs  in  a  restricted  EMCON  status  and  the  remaining  seven 
emitting. 

4.  Anticipated  Results 

The  hypothesized  results  were  that  friendly  (Blue)  forces  would  perform  better 
when  the  following  occurred: 

•  Information  or  intelligence  reports  were  complete. 

•  The  ship’s  combat  team  was  experiencing  low  workload  conditions. 

•  Incoming  attacks  were  imcoordinated  (sequential)  vice  coordinated  (concurrent). 
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B.  SCOPE  OF  THE  EXPERIMENT 


The  complexity  of  the  scenarios  for  this  experiment  was  limited  due  to  time 
constraints  and  the  varied  backgrounds  and  limited  operational  experience  of  some  of  the 
subjects.^  Blue  Force  resources  were  initialized  in  the  same  manner  for  every  scenario  and 
for  every  subject.  This  standardization  was  done  to  reduce  the  risk  of  subjects  with 
operational  experience  gaining  an  advantage  by  relying  on  their  expert  knowledge  to 
position  their  forces.  In  addition,  certain  artificialities  were  accepted  in  the  interest  of 
practicality  and  usability.  For  instance,  the  wargame  was  run  at  a  speed  of  4:1  (i.e.,  4 
seconds  of  simulated  time  =  1  second  of  real  time).  Therefore  a  152  minute  simulated  event 
could  be  compressed  into  38  minutes  of  real  time.  All  scenarios  were  constmcted  similarly 
except  to  allow  for  the  systematic  introduction  of  both  levels  of  all  factors.  Scenarios  were 
constructed  to  appear  different  to  the  subjects  without  affecting  essential  traits.  This 
diminished  the  chance  of  subjects  relying  on  prior  scenario  knowledge  to  game  the 
scenario  instead  of  making  decisions  based  on  current  scenario  developments. 

C.  ORGANIZATION 

The  remainder  of  the  thesis  is  organized  as  follows:  Chapter  II  introduces  the 
design  of  the  experiment  including  the  set-up,  subjects,  hypotheses,  assumptions,  statistical 
design  of  the  experiment,  and  measures.  Chapter  III  describes  the  data,  how  the  raw  data 
were  collected  and  transformed,  the  data  coding  scheme,  and  data  reduction  phase.  Chapter 
IV  presents  the  data  analysis  plan,  results  of  the  data  analysis  and  demographic  information 
on  the  subjects.  Chapter  V  presents  conclusions  including  results  of  hypotheses  tests  and 
their  real  world  meanings.  Finally,  Chapter  VI  offers  recommendations  to  change,  improve 
upon,  and  extend  this  experiment  for  the  future. 


2  The  experiment  had  to  be  set-up,  played,  and  recorded  in  the  50  minute  class  period. 
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II.  EXPERIMENTAL  DESIGN 


This  chapter  discusses  experimental  set-up  and  presents  the  seven  hypotheses  that 
are  associated  with  this  two-cubed  experiment.  In  addition  the  assumptions,  statistical 
design,  and  measures  of  the  experiment  are  explained. 

A.  SET  -  UP 

In  this  section  the  software,  hardware,  and  subjects  are  discussed.  B&R  was  chosen 
because  of  the  ease  and  speed  with  which  scenarios  could  be  generated  and  modified.  It 
addition,  it  has  a  superior  graphical  user  interface  and  artificial  intelligence  which  allows 
pre-scripted  Red  forces  to  deviate  fi'om  the  script  in  Batman  and  behave  opportunistically. 
Finally,  B&R  has  an  automatic  data  collection  capability  which  is  very  useful  in  the  data 
collection  phase. 

1.  Software  and  Hardware 

B&R  software  was  obtained  fi'om  the  creator  of  the  wargame.  Dr.  Pat-Anthony 
Federico,  at  the  Naval  Personnel  Research  and  Development  Center  (NPRDC),  San  Diego. 
It  is  written  in  the  “C”  programming  language.  B&R  wzis  installed  in  the  NPS  Systems 
Technology  Lab  (STL)  on  five  Sun  Sparc-20  stations  and  runs  in  a  UNIX  Solaris  2.5 
environment.  All  scenarios  were  created  in  Robin  on  one  Sparc  station  and  ported  to  the 
other  four  to  ensure  standardization.  Each  subject  executed  each  scenario  in  Batman 
independently  of  other  subjects.  B&R  is  not  capable  of  running  in  a  multi-player 
networked  configuration. 

2.  Subjects 

Sixteen  officers  participated  as  subjects  in  the  experiment.  Before  the  experimental 
trials  commenced,  all  were  given  a  50  minute  standardization  overview  brief  of:  SAG 
tactics,  platform  capabilities,  sensor  and  weapon  performance  parameters,  weapon  status 
and  weapon  posture  meanings,  and  B&R  rules.  This  was  done  for  standardization  purposes 
and  in  an  effort  to  mitigate  learning  effects.  Each  subject  was  administered  two  practice 
trials  followed  by  the  eight  experimental  trials  in  which  data  was  collected.  During  the 
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practice  trials,  subjects  received  one-on-one  instruction  on  B&R  from  trained  proctors.  All 
trials  were  completed  during  a  ten  day  period  due  to  course  requirement  constraints  at  NFS. 
Subjects  were  limited  to  three  trials  per  day  to  preclude  fatigue.  Subjects  were  not  permitted 
to  discuss  their  trials  with  other  subjects  before,  during,  or  after  trials.  The  experiment  was 
conducted  in  a  controlled  environment  with  physical  dividers  placed  on  both  sides  of  each 
subject’s  computer  station.  Demographic  information  was  collected  on  all  of  the  subjects 
for  future  stratification  analysis  (see  Chapter  IV.B.4.  for  more  information). 

B.  HYPOTHESES 

Ten  performance  measures  (PMs)  or  dependent  variables  were  recorded  for  each 
trial.  For  a  description  of  the  PMs  see  Section  G.  The  following  hypotheses  were  tested  for 
each  PM: 


•  Hoj :  Information  Completeness  has  no  impact  on  SAG  performance. 

•  H,j:  Information  Completeness  impacts  SAG  performance. 

•  Ho  Work  Load  has  no  impact  on  SAG  performance. 

•  H32:  Work  Load  impacts  SAG  performance. 

•  H03:  FPB  C^  has  no  impact  on  SAG  performance. 

•  H33:  FPB  C^  impacts  SAG  performance. 

•  H04:  The  combined  effect  of  Information  Completeness  and  Work  Load  has  no 
impact  on  SAG  performance. 

•  H34:  The  combined  effect  of  Information  Completeness  and  Work  Load  impacts 
SAG  performance. 

•  Hos:  combined  effect  of  Information  Completeness  and  FPB  C^  has  no 

impact  on  SAG  performance. 

•  Hjg:  The  combined  effect  of  Information  Completeness  and  FPB  C^  impacts 
SAG  performance. 


The  combined  effect  of  Work  Load  and  FPB  has  no  impact  on  SAG 
performance. 

•  Hj,g;  The  combined  effect  of  Work  Load  and  FPB  impacts  SAG  performance. 

•  Hp,:  The  combined  effect  of  Information  Completeness,  Work  Load  and  FPB 
has  no  impact  on  SAG  performance. 

•  Hav;  The  combined  effect  of  Information  Completeness,  Work  Load  and  FPB  C^ 
impacts  SAG  performance. 

The  significance  level  for  null  hypotheses  rejection  was  set  at  alpha  -  0.05.  For 
convenience,  these  hypotheses  are  repeated  in  Appendix  I. 

C.  ASSUMPTIONS 

Four  assumptions  were  made  at  the  outset  of  the  trials: 

•  Scenario  outcomes  represent  the  subject’s  ability  to  act  on  or  react  to  the  cues 
displayed  by  Batman. 

•  The  ten  performance  measures  (dependent  variables)  chosen  from  Batman ’s 
automatic  data  collection  capability  adequately  identified  the  impact  or 
influence  of  the  factors  (independent  variables)  and  their  interactions  on  the 
outcome  of  each  scenario. 

•  The  subjects’  performance  will  be  normally  distributed  and  all  populations  will 
have  the  same  variance. 

•  The  scenarios  are  designed  to  measure  the  subjects’  performance  over  the 
dynamic  range  of  the  performance  curve. 

D.  STATISTICAL  DESIGN  OF  THE  EXPERIMENT 

The  experiment  was  counter-balanced  to  negate  any  learning  effect  remaining  after 
the  practice  trials.  The  two-cubed  design  of  the  experiment  yielded  eight  unique  scenarios. 
Using  16  subjects  and  a  completely  randomized  block  design  with  2  replications  yielded  an 
8x16  output  matrix.  To  construct  the  design  matrix,  the  scenario  type,  order  of  scenario 
presentation  and  subject’s  assignment  to  a  scenario  presentation  were  all  randomized.  This 
triple  randomization  was  accomplished  using  one  8x8  Latin  Square  for  rows  1-8;  the 
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Latin  Square  was  composed  of  the  letters  a-h.  A  duplicate  Latin  Square  was  then  inverted 
and  used  to  create  rows  9-16.  LFsing  ‘APL2’  programming  language,  unique  random 
numbers  were  randomly  assigned  to  the  eight  columns  and  then  ordered.  The  same  process 
was  carried  out  for  the  16  rows.  Since  the  initial  scenario  numbering  scheme  was  a  flmction 
of  the  original  order  of  generation  and  included  numbers  greater  than  8,  the  scenarios  were 
renumbered  arbitrarily  from  1-8.  Those  numbers  were  then  randomized  and  assigned  the 
letters  a-h.  As  a  last  step,  we  used  a  manual  random  number  generator  to  randomly  assign 
the  subjects  to  the  rows.  B&R  has  a  utility  function  enabling  the  administrator  to  assign 
subjects  to  classes,  scenarios  to  tests,  and  finally,  classes  to  tests.  Assignments  were  based 
on  the  outcome  of  the  counter-balancing  process.  This  function  minimized  the  chance  of 
error  inherent  with  administering  160  trials  (32  training  and  128  experimental  trials). 

E.  MEASURES 

B&R’s  automatic  data  collection  feature  allowed  81  performance  measures  (PMs) 
or  dependent  variables  to  be  collected  after  each  trial.  The  81  PMs  are  divided  into  two 
broad  categories:  Blue  Forces’  performance  vs.  Red  Forces,  and  Red  Forces’  performance 
vs.  Blue  Forces.  Each  of  these  two  broad  categories  is  further  divided  into  3  sub-categories: 
air  forces  vs.  surface,  surface  forces  vs.  surface,  and  combined  air/surface  forces  vs.  surface. 
Not  all  81  PMs  were  applicable  to  this  experiment.  After  a  winnowing  process  the  data  was 
reduced  to  10  PMs.  Of  the  10  PMs  selected  the  first  five  measure  Blue  Forces’ 
performance  and  the  second  five  measure  Red  Forces’  performance.  The  PMs  used  are 
defined  as  follows: 

•  BAFP 1:  FPBs  detected  by  Blue  Air  (%). 

•  BAFP  2:  Average  range  (measured  from  nearest  Blue  force  ship)  at  which 
FPBs  were  first  detected  by  Blue  Air  (nm.). 

•  BSP  3:  FPB’s  destroyed  by  Blue  Surface  (%). 

•  BSP  4:  Average  range  (measured  from  nearest  Blue  force  ship)  at  which  FPBs 
were  destroyed  by  Blue  Surface  (nm.). 
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•  BAFP 12:  Number  of  TAO  weapon  assignments  (count)/ 

•  A 1:  Blue  Air  detected  by  the  FPBs  (%). 

•  SI:  Blue  Surface  detected  by  the  FPBs  (%). 

•  A  2:  Average  range  (measured  from  nearest  FPB)  at  which  Blue  Air  was  first 
detected  by  the  FPBs  (nm.). 

•  S  2:  Average  range  (measured  from  nearest  FPB)  at  which  Blue  Surface  was 
first  detected  by  the  FPBs  (nm.). 

•  A3:  Blue  Air  destroyed  by  the  FPBs  (%). 


3  Blue  forces  had  to  utilize  the  TAO  function  in  order  to  fire  their  weapons  and  engage  Red  forces.  This  number 
represents  the  number  of  salvos  launched  by  the  operator  per  trial. 
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III.  DATA  DESCRIPTION 


This  chapter  discusses  the  data  collection  and  transformation  process,  the  data 
coding  scheme,  and  the  data  reduction  phase. 

A.  DATA  COLLECTION  AND  TRANSFORMATION 

Although  B&R  calculated  values  for  the  PMs  automatically,  hardware  limitations 
prohibited  the  automatic  transfer  of  data  to  a  file.  Instead,  each  subject  manually  recorded 
their  performance  data  displayed  on  the  monitor  to  the  PM  data  collection  sheet  at  the  end 
of  each  scenario.  The  large  number  of  subjects  participating  at  a  given  time  precluded 
quality  control  while  the  subjects  were  recording  their  data.  The  information  was  then 
manually  entered  into  Excel®.  Before  any  data  manipulations  were  performed,  the  data 
were  put  through  a  rigorous  quality  control  process  to  check  for  data  input  errors.  Data 
reduction  was  then  performed  and  the  data  ported  from  Excel  into  Minitab®  for  statistical 
analysis.  Once  the  data  were  in  Minitab  it  was  checked  ageiin  for  accuracy. 

B.  EXAMPLES  OF  RAW  DATA 

Appendix  A  contains  an  example  of  a  subject’s  PM  data  collection  sheet  (a  PM 
sheet  was  generated  after  each  trial).  Notice  that  all  81  possible  PMs  were  recorded  on  the 
PM  data  collection  sheet.  This  was  done  so  the  data  could  be  recorded  quickly  and 
accurately.  As  stated  previously,  only  10  PMs  were  actually  used  in  the  subsequent 
analysis.  Appendix  B  contains  a  sample  of  the  master  data  spreadsheet.  The  spreadsheet 
contains  data  for  each  of  the  16  subjects  organized  by  trial  number,  scenario  number,  and 
PMs  results.  Scenario  numbers  are  described  in  the  next  section. 

C.  DATA  CODING  SCHEME 

Each  of  the  eight  scenarios  was  uniquely  described  by  the  level,  high  or  low,  of  each 
of  the  three  factors:  Information  Completeness  ( I ),  Work  Load  (  L  ),  and  FPB  (  C  ). 
This  information  was  coded  as  a  three  digit  number  with  each  digit  either  a  ‘2’  (high)  or  a 
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‘  1  ’  (low).  Once  the  data  from  a  PM  sheet  was  entered  into  Excel,  the  scenario  from  which 
it  was  generated  was  referred  to  by  this  three  digit  code  (see  Appendix  B,  columns  3, 4,  and 
5).  For  example,  the  scenario  with  I,  L,  and  C  at  the  high  level  is  coded  222.  The  scenario 
with  all  factors  at  the  low  level  is  coded  111.  Referencing  sub-section  I.  A.4.,  we  anticipated 
scenario  21 1  (high  or  complete  information,  low  work  load,  low  FPB  tactical  coordination) 
would  produce  the  best  results  for  Blue  forces. 

D.  DATA  REDUCTION 

Initial  data  reduction  was  performed  by  reducing  the  81  PMs  which  B&R  was 
capable  of  producing  for  this  experiment  to  the  ten  projected  to  be  most  relevant  (see 
section  lI.E.).  This  was  accomplished  before  the  data  collection  phase.  B&R  automatically 
converts  a  subject’s  performance  to  percentages,  counts,  and  average  distances  and  then 
lists  them  as  performance  measures.  After  the  data  collection  phase  the  ten  selected  PMs 
were  checked  to  ensure  sufBcient  variability  existed  to  warrant  continuing  with  the  analysis 
phase.  Most  of  the  PMs  chosen  produced  good  distributions. 
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IV.  ANALYSIS 


This  chapter  discusses  the  overall  analysis  plan  and  presents  the  results  of  the 
analysis.  The  information  in  this  chapter  is  quite  detailed  and  necessary  for  a  complete 
understanding  of  the  results.  A  summary  of  findings  is  presented  in  the  Conclusions 
(Chapter  V). 

A.  ANALYSIS  PLAN 

Initial  data  analysis  utilized  frequency  plots  to  obtain  a  pictorial  representation  of 
the  128  data  points  per  PM  (16  subjects  x  8  scenarios).  This  established  an  initial  feel  for 
the  data  and  provided  insight  for  fiirther  detailed  analysis.  The  shape  of  some  of  the  plots 
may  call  into  question  our  initial  assumption  of  normality.  Next,  box  plots  were  used  to 
examine  the  three  factors’  individual  impacts  on  the  PMs.  After  a  working  understanding 
of  the  data  was  obtained,  three  factor  MANOVA  was  used  as  a  screening  device,  and  since 
not  all  PM  distributions  appeared  normal,  parametric  and  non-parametric  ANOVA, 
interaction  plots  and  residual  plots  were  used  to  conduct  a  detailed  analysis.  A  significance 
level  of  a  =  0.05  was  used  to  test  all  null  hypotheses.  When  the  data  did  not  appear  normal, 
non-parametric  ANOVAs  were  conducted  over  the  factors  suggested  by  MANOVA, 
parametric  ANOVA,  or  the  box  plots. 

B.  RESULTS  OF  ANALYSIS 

Detailed  results  of  the  analysis  presented  in  this  section  include  frequency  plots  and 
the  corresponding  descriptive  statistics,  box  plots,  ANOVA,  two-way  interaction  plots,  and 
residual  plots.  In  addition,  the  results  of  preliminary  analysis  performed  on  the 
demographic  information  is  included. 

1.  Frequency  Plots  and  Descriptive  Statistics  (see  Appendix  C  for  plots) 

BAFP  1  (FPBs  detected  by  Blue  Air  (%)):  Almost  all  FPBs  were  detected  by  Blue 
Air  throughout  the  experiment.  This  is  not  surprising;  Blue  Forces  had  five  helicopters  to 
conduct  Surface  Search  Surveillance  and  Coordination  (SSSC). 


13 


BAPP  2  (Average  range  at  which  FPBs  were  first  detected  by  Blue  Air  (nm.)):  The 
average  range  that  Blue  Air  first  detected  the  FPBs  was  71  nm.  measured  from  the  nearest 
Blue  force  ship. 

BSP  3  (FPB’s  destroyed  by  Blue  Surface  (%)):  Overall  75%  of  the  FPBs  were 
destroyed.  BAFP  1  and  BSP  3  plots  suggest  that  not  all  FPBs  were  destroyed  even  though 
almost  all  were  detected.  This  is  probably  because  Blue  Forces  expended  their  limited 
Harpoon  inventory  (8  weapons  per  ship)  or  were  not  able  to  position  themselves  within  gun 
range  due  to  the  time  constraints  of  the  experiment. 

BSP  4  (Average  range  at  which  FPBs  were  destroyed  by  Blue  Surface  (nm.)):  Blue 
Surface  destroyed  the  FPBs  at  an  average  range  of  30  nm.  measured  from  the  nearest  Blue 
force  ship  (Harpoon  range  is  5-75  nm.;  gun  range  is  1-15  nm.). 

BAFP  12  (Number  of  TAO  weapon  assignments  (count)):  The  average  number  of 
TAO  weapon  assignments  per  game  was  approximately  19  (i.e.,  the  number  of  salvos 
laimched  by  the  operator  per  trial). 

A  1  (Blue  Air  detected  by  the  FPBs  (%)):  All  Blue  Air  platforms  were  detected  by 
the  FPBs  in  most  of  the  trials.  Blue  Air  assets  were  required  to  come  within  the  FPBs’ 
surveillance  envelopes  to  positively  identify  contacts  as  hostile. 

S  1  (Blue  Surface  detected  by  the  FPBs  (%)):  Overall  23.19%  of  Blue  Surface 
platforms  were  detected  by  the  FPBs.  This  is  due  in  part  to  EMCON  restrictions  used  by 
some  FPBs. 

A  2  (Average  range  at  which  Blue  Air  was  first  detected  by  the  FPBs  (nm.)):  The 
average  range  that  the  FPBs  first  detected  Blue  Air  was  45.3  nm.  Measured  from  the  nearest 
FPB. 

S  2  (Average  range  at  which  Blue  Surface  was  first  detected  by  the  FPBs  (nm.)): 
The  average  range  at  that  the  FPBs  detected  Blue  Surface  was  17.4  nm.  Measured  from  the 
nearest  FPB.  By  combining  plots  SI  and  S2,  it  is  apparent  that  the  FPBs  and  Blue  Surface 
platforms  operated  at  distances  greater  than  their  surveillance  areas.  The  initial  separation 
of  Blue  Surface  and  the  FPBs  at  the  start  of  the  game  was  great  enough  so  that  neither 
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would  be  in  the  other’s  sensor  range.  This  was  done  to  force  Blue  to  rely  on  the  helicopters 
and  intelligence  provided  to  locate  the  FPBs. 

A  3:  (Blue  Air  destroyed  by  the  FPBs  (%)):  Overall  21%  of  Blue  Air  was  destroyed 
by  the  FPBs.  Blue  Air  was  susceptible  to  the  FPBs’  surface  to  air  missiles  while 
conducting  visual  identification.  Electronic  Surveillance  Measures  (ESM)  information 
when  available  to  Blue,  may  have  made  subjects  more  vigilant  when  operating  air  assets  in 
the  vicinity  of  unknown  surface  platforms.  Without  this  information,  the  percentage  of 
Blue  Air  destroyed  could  have  been  higher. 

2.  Box  Plots  (see  Appendix  D  for  plots) 

As  described  earlier,  the  experiment  examined  the  effects  of  three  factors  at  two 
levels.  Box  plots  were  used  to  determine  if  there  were  significant  differences  between  the 
distributions  of  the  PMs  for  the  two  levels  of  treatment  for  each  of  the  three  factors.  Box 
plots  present  a  pictorial  representation  of  the  key  features  of  the  data:  median,  first  and  third 
quartiles,  and  spread.  Significant  differences  in  these  features  could  indicate  differences  in 
the  location  or  spread  of  the  PM  distribution  when  factor  levels  are  altered.  Within  the 
context  of  these  box  plots,  significant  implies  a  non-trivial  visual  difference  in  the  box  plot 
with  respect  to  the  median  and  the  first  and  third  quartile.  If  differences  were  identified, 
ANOVA  was  conducted  to  gain  additional  insight. 

Four  sets  of  box  plots  were  generated  for  each  PM.  The  first  three  correspond  to 
individual  factors.  The  boxes  labeled  with  a  1  represent  the  64  data  points  corresponding  to 
the  low  level  of  the  factor,  whereas  those  labeled  with  a  2  represent  data  for  the  high  level 
of  each  factor.  The  fourth  set  of  box  plot  graphics  contain  eight  box  plots  representing  the 
eight  unique  scenarios  (each  scenario’s  box  plot  contains  16  data  points).  It  is  coded  in 
accordance  with  the  data  coding  scheme  described  earlier.  The  box  plots  were  used  as  a 
tool  to  help  corroborate  or  refute  the  rejection  of  a  null  hypotheses  and  to  gain  insight  into 
the  nature  of  any  effects. 
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3.  ANOVA  /  Interaction  and  Residuals  Plots  (see  Appendices  E  and  F  for 

plots) 

This  sub-section  builds  on  the  results  of  the  analysis  based  on  the  frequency  plots 
and  box  plots  and  uses  the  additional  statistical  tools  described  earlier  to  make  the  case  for 
acceptance  or  rejection  of  the  null  hypotheses  (see  Appendix  I)  for  each  PM. 

BAFP  1  (FPBs  detected  by  Blue  Air  (%)):  The  Normal  Plot  of  the  ANOVA 
Residuals  (NPR)  suggests  the  data  does  not  behave  normally.  This  possibility  is 
corroborated  by  the  Histogram  of  the  ANOVA  Residuals  Plot  (HRP)  and  the  ANOVA 
Residuals  vs.  ANOVA  Fits  Plot  (RFP).  ANOVA  yields  a  p-value  for  C  of  0.000.  This 
implies  that  Hqj  should  be  rejected,  however,  due  to  the  non-normality  of  the  data  we  can 
not  reject  H03  based  on  ANOVA.  The  Kruskal-Wallis  (K-W)  non-parametric  ANOVA, 
which  does  not  require  normality,  was  conducted  and  also  yielded  a  p-value  of  0.000. 
Based  on  the  combined  result,  we  reject  H03.  It  makes  sense  that  the  level  of  would  have 
an  impact  on  Blue  Air’s  ability  to  detect  the  FPBs.  The  difference  in  means  for  Cl  and  C2 
is  98.09%  detected  and  90.10%  detected  respectively.  Nothing  else  of  significance  was 
found. 

BAFP  2  (Average  range  at  which  FPBs  were  first  detected  by  Blue  Air  (nm.)):  The 
NPR,  HRP  and  the  RFP  all  indicate  that  the  data  is  normal.  ANOVA  yields  a  p-value  of 
0.001  and  0.000  for  L  and  C  respectively.  The  box  plots  show  a  median  value  for  BAFP2 
that  is  approximately  the  same  for  LI  and  L2,  however,  the  mean  value  of  detection  is  more 
than  3  nm.  greater  when  the  workload  is  low  (73.02  vice  69.63nm.).  For  these  reasons  we 
reject  H02.  H03  is  also  rejected,  however,  the  operational  rational  is  not  so  intuitive.  The 
boxplots  show  a  significant  difference  in  both  mean  and  median  with  Blue  Air  detecting  the 
FPBs  further  out  at  C2  probably  because  there  was  more  time  between  engagements  to 
move  own  forces  out.  The  p-value  for  the  LxC  interaction  is  0.059.  Although  this  is  very 
close  to  the  significance  level,  the  null  hypothesis  for  H04  will  not  be  rejected.  The 
Interaction  Plots  (IP)  for  LxC  reveals  little  interaction.  The  p-value  for  IxLxC  is  0.003. 
This  leads  us  to  reject  H07 
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BSP  3  (FPB’s  destroyed  by  Blue  Surface  (%)):  The  p-values  based  on  ANOVA 
suggests  not  rejecting  any  null  hypotheses  except  of  H03  (p-value  =0.000).  NPR,  HRP,  and 
the  RFP  all  show  the  data  to  deviate  from  the  normal.  Hence,  we  will  not  base  our  decision 
on  ANOVA  but  instead  on  K-W.  K-W  yields  a  p-value  of  0.000  for  H03;  on  that  evidence 
we  reject  H03.  For  H03,  the  box  plots  for  Cl  and  C2  (medians  =  88.89%  and  66.67% 
respectively)  indicate  that  fewer  FPB’s  were  killed  by  Blue  forces  when  under  a 
coordinated  attack  (C2).  This  supports  our  primary  expectation  that  Blue  forces  will 
perform  worse  when  attacked  in  mass  by  FPBs  using  COTS  Cl  The  IPs  show  no 
interactions  exist  for  IxL,  IxC,  and  LxC  (p-values  are  0.460, 0.460, 0.715,  respectively) 

BSP  4:  (Average  range  at  which  FPBs  were  destroyed  by  Blue  Surface  (nm.)): 
NPR,  HPR,  and  the  RFP  all  indicate  normal  behavior  of  the  data.  The  p-value  for  I  (0.047) 
indicates  Hq,  should  be  rejected.  The  descriptive  statistics  indicate  a  significant  difference 
between  the  means  II  and  12  (28.5nm  and  31.78nm  respectively).  Therefore  Ho,  is  rejected. 
We  expect  that  with  good  information  Blue  forces  will  kill  the  enemy  fiirther  out.  For  L 
and  C  the  p-values  also  make  the  case  to  reject  H02/03  (0.001  and  0.002  respectively).  The 
single  factor  box  plot  for  L  shows  that  Blue  destroyed  the  FPBs  further  out  when  the  load 
was  light  as  we  expect.  The  impact  that  factor  C  displays  follows  from  BAFP2:  since  Blue 
Air  detected  the  FPBs  further  out  at  C2,  the  FPBs  destroyed  were  done  so  at  a  greater 
distance.  This  makes  operational  sense:  if  the  enemy  is  detected  further  away  then  it  be 
destroyed  at  or  near  the  outer  limit  of  the  weapon’s  range.  For  all  these  reasons  we  reject 
Ho2/o3'  yields  a  p-value  of  0.011.  The  IP  also  suggests  rejecting  H05  by  showing  a 
strong  interaction  between  I  and  C  therefore  we  reject  H05.  This  synergy  between  I  and  C  is 
most  apparent  when  C  is  high  and  can  be  explained  as  follows:  when  the  enemy’s  attack 
plan  is  organized  it  is  easier  to  predict,  from  present  positional  information,  future 
movements  based  on  extrapolation  and  to  detect  the  enemy  earlier  (i.e.,  at  a  greater  range) 
than  when  the  attack  is  less  coordinated  (more  random).  IxL,  LxC  and  IxLxC  all  have  p- 
values  »0.05.  The  IP  for  IxL  and  LxC  confirms  the  acceptance  of  H04/06  by  showing  little 
to  no  interaction. 
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BAFP  12  (Number  of  TAO  weapon  assignments  (coimt)):  NPR,  HPR,  and  the  RFP 
all  indicate  normal  behavior  of  the  data.  The  p-value  for  C  is  0.000.  The  single  factor  box 
plot  for  C  reveals  a  significant  difference  between  the  median  number  of  TAO  weapon 
assignments  for  Cl  and  C2  (23  and  18  count  respectively).  A  likely  reason  for  fewer 
weapon  assignments  being  made  when  Blue  was  under  a  coordinated  attack  follows  from 
BAFPl:  since  fewer  FPBs  were  detected  when  Blue  was  under  a  coordinated  attack,  fewer 
could  be  targeted.  For  these  reasons  we  reject  H03.  No  other  p-values  suggest  rejecting  a 
null  hypothesis. 

A  1  (Blue  Air  detected  by  the  FPBs  (%)):  NPR,  HPR,  and  the  RFP  all  indicate  non- 
normal  behavior  of  the  data.  ANOVA  yields  a  p-value  of  0.012  for  factor  C.  K-W  yields  a 
p-value  of  0.02.  The  single  factor  box  plot  for  C  shows  that  the  FPBs  detected  far  fewer 
helicopters  when  fiiey  were  conducting  a  coordinated  attack  (this  was  expected  since  seven 
of  the  nine  FPBs  were  in  EMCON  for  C2).  For  these  reasons  Hqj  is  rejected.  ANOVA  for 
LxC  (Hos)  yields  0.032.  The  IP  for  LxC  shows  that  more  helicopters  were  detected  by  Red 
when  the  load  for  Blue  was  low  (for  both  high  or  low  C).  The  possible  reason:  when  the 
load  was  low  the  helicopters  were  more  accurately  vectored  to  the  FPBs.  The  IP  for  LxC 
also  shows  that  when  the  load  was  high  the  percentage  of  helicopters  detected  by  Red  was 
about  the  same  regardless  of  the  level  of  C.  For  this  we  can  find  no  clear  operational 
explanation. 

S  1 :  (Blue  Surface  detected  by  the  FPBs  (%)):  NPR,  HPR,  and  the  RFP  all  indicate 
normal  behavior  of  the  data,  however,  the  initial  frequency  plot  of  S  1  suggests  the  data 
may  not  be  useful  because  there  are  only  three  Blue  ships  to  detect.  ANOVA  yields  a  p- 
value  of  0.003  for  C.  The  single  factor  box  plot  for  C  registers  a  significant  difference 
between  Cl  and  C2  (medians  =  33.0%  and  17.0%  respectively)  confirming  the  low  p- 
value’s  indication  that  we  reject  Hqj  (when  C  was  high  most  of  the  FPBs  were  in  EMCON). 
IxC  p-value  is  0.047.  The  IP  also  supports  rejecting  H05,  however,  the  full  operational 
explanation  of  the  interaction  displayed  is  unclear.  When  C  is  low  the  percentage  of  Blue 
Svuface  detected  by  the  FPB’s  increases  when  Blue  has  complete  information  (Blue  closes 
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the  distance  with  Red  more  promptly  with  complete  information  therefore  Red  also 
becomes  aware  of  Blue’s  presence).  When  C  is  high  the  opposite  occurs.  Perhaps  this  is 
because  there  are  more  unintentional  encounters  between  Blue  and  the  FPBs  when 
information  is  incomplete. 

A  2:  (Average  range  at  which  Blue  Air  was  first  detected  by  the  FPBs  (nm.)): 
NPR,  HPR,  and  the  RFP  all  indicate  normal  behavior  of  the  data.  ANOVA  yields  only  one 
p-value  below  the  cutoff:  LxC  =  0.000.  The  IP  for  LxC  also  suggests  rejecting  Hog.  The 
operational  explanation  of  the  nature  of  the  interaction  between  LxC  is  imclear. 

S  2  (Average  range  at  which  Blue  Surface  was  first  detected  by  the  FPBs  (nm.)): 
During  24  of  the  128  trials  Blue  Surface  was  never  detected  by  the  FPBs  (this  occurred 
seven  times  at  Cl  and  17  times  at  C2),  unfortunately,  that  left  the  data  set  incomplete  for 
this  parameter.  The  initial  frequency  plot  and  the  box  plots  indicate  that  the  data  is  not 
normal.  The  p-value  for  K-W  with  C  as  the  factor  was  0.081 .  The  most  we  can  say  is  that 
the  mean  value  at  which  the  FPBs  detected  Blue  Surface  is  17.42  nm.  measured  from  the 
nearest  FPB.  The  mean  values  for  Cl  and  C2  are  17.7  and  17.08,  respectively. 

A  3  (Blue  Air  destroyed  by  the  FPBs  (%)):  The  initial  frequency  plot  indicates  the 
data  is  skewed  to  the  right.  The  NPR  and  RFP  also  indicate  that  the  data  is  non-normal. 
ANOVA  yields  a  p-value  of  0.002  for  C.  K-W  for  C  yields  0.053;  not  quite  small  enough 
to  reject  Hqj.  The  mean  percentage  of  helicopters  destroyed  for  Cl  and  C2  (15.4%  and 
26.9%  respectively)  indicates  that  when  the  FPBs  conducted  a  coordinated  attack  more 
helicopters  were  destroyed.  This  may  follow  from  the  fact  that,  although  we  could  not 
reject  H03  for  A2  (ANOVA  p-value=0.164),  the  single  factor  box  plot  for  C  of  that  factor 
revealed  a  similar  finding  with  the  FPBs  detecting  Blue  air  further  out  when  conducting  a 
coordinated  attack  versus  a  non-coordinated  attack  (medians  =  42  and  47nm.  respectively). 


4  Analyzing  the  24  times  that  Blue  Surface  was  totally  undetected  using  a  chi-square  test  leads  to  rejecting  the  null 
hypothesis  that  Blue  Surface  is  equally  likely  to  go  totally  undetected  for  both  Cl  and  C2  (p-value  of 0.024).  The 
estimated  probabilities  of  Blue  Surface  going  totally  undetected  are  Cl ;  0.1 1  and  C2: 0.27. 
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It  also  makes  sense  that  we  will  lose  more  scouts  when  the  FPBs  are  in  an  EMCON  status 
because  Blue  Air  must  enter  the  FPBs  weapons  envelope  to  positively  identify  them.  The 
IP  for  LxC  reveals  when  the  FPBs  were  coordinated  in  their  attack  and  Blue  forces  work 
load  was  high,  more  helicopters  were  destroyed.  We  already  know  C  has  an  impact,  but 
this  indicates  that  Blue  Air  assets  were  not  managed  as  carefully  or  prudently  when  the 
work  load  on  Blue  was  high. 

4.  Demographic  Information 

Of  the  16  subjects,  eight  were  USN,  five  were  USMC,  two  were  USAF,  and  one 
was  USA.  Six  of  the  eight  naval  officers  were  surface  warfare  officers.  Thirteen  of  the  16 
subjects  were  students  in  the  Command,  Control,  and  Communications  (C^)  curriculum  at 
NPS.  Years  of  service  ranged  from  5-16  years.  A  demographic  questionnaire  was 
completed  by  each  subject  during  the  overview  brief;  the  questions  were  as  follows: 

1 .  Name 

2.  Rank  /  Branch  of  Service 

3.  Designator  /  MOS  Description 

4.  Number  ofYears  on  Active  Duty 

Subjects  were  asked  to  rate  on  a  scale  of  from  one  (poor)  to  ten  (expert)  their: 

5 .  Level  of  Proficiency  with  Navy  Tactical  Data  System  (NTDS)  symbology 

6.  Level  of  Proficiency  with  Simulation  /  Wargames 

7.  Level  of  Proficiency  as  Tactical  Action  Officer  (TAO) 

8.  Level  of  Computer  Proficiency 

The  answers  to  questions  four  through  eight  are  discussed  below: 

•  The  distribution  of  the  years  of  service  was  skewed  toward  more  senior  officers 
(see  Appendix  G,  Figure  1  for  details). 

•  The  question  on  NTDS  symbology  was  asked  because  the  symbology  used  in 
B&R  is  very  similar  to  NTDS  symbology;  it  was  presumed  that  familiarity  with 
similar  symbology  would  give  a  subject  an  advantage  in  adjusting  to  the 
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wargame.  As  expected,  the  distribution  of  the  responses  to  this  question  were 
bimodal  with  ten  of  the  sixteen  subjects  evenly  split  between  the  top  end  of  the 
scale  and  the  bottom  end  (see  Appendix  G,  Figure  2  for  details). 

•  Experience  with  wargames  was  chosen  early  on  as  a  factor  that  could  influence 
a  subject’s  performance.  The  responses  to  the  question  regarding  wargame 
experience  appears  to  approach  a  normal  distribution  with  responses  skewed 
slightly  toward  more  experience  (see  Appendix  G,  Figure  3  for  details). 

•  Another  factor  that  was  anticipated  to  impact  performance  was  proficiency  as  a 
TAG.  Since  a  TAG  qualification  is  specific  to  SWG  only,  we  assumed  that  the 
distribution  would  be  skewed  toward  the  more  proficient  side  since  the 
participant  group  contained  a  wide  range  of  warfare  specialties.  A  full  fifty 
percent  of  respondents  claimed  the  lowest  level  on  the  proficiency  scale  (see 
Appendix  G,  Figure  4  for  details). 

•  Finally,  the  data  pertaining  to  the  level  of  computer  proficiency  was  evaluated. 
The  data  for  this  question  appears  to  have  a  normal  distribution  skewed  toward 
the  less  proficient  end,  vrith  seventy  five  percent  of  the  respondents  claiming 
proficiency  above  the  midpoint  on  the  scale  (see  Appendix  G,  Figure  5  for 
details). 
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V.  CONCLUSIONS 


This  chapter  discusses  hypotheses  results  and  provides  a  summary  of  the  most 
significant  results.  In  addition,  corresponding  real  world  interpretations  are  offered.  The 
previous  chapter  was  predominantly  organized  by  PMs.  This  chapter  is  organized  by  the 
factors  and  interactions  (hypotheses)  and  focuses  on  significant  findings.  Findings  are  also 
condensed  and  presented  in  table  form  in  Appendix  H  (Null  Hypotheses  Rejection 
Summary). 

A.  HYPOTHESES  RESULTS 

All  three  factors  and  three  out  of  four  of  their  interactions  produced  significant 
effects,  although  not  for  all  PMs.  All  significant  effects  are  described  below. 

1.  Information  Completeness 

Hoi  had  a  significant  impact  on  one  PM:  The  average  range  at  which  the  FPBs  were 
destroyed  by  Blue  Surface  (BSP  4);  p  =  0.047,  31.8  nm.  when  information  was  complete, 
and  28.5  nm.  when  information  was  incomplete.  With  complete  information  Blue  forces 
were  able  to  kill  the  enemy  over  3  nm.  further  out  than  with  incomplete  information. 

2.  Work  Load 

Ho2  had  a  significant  impact  on  two  PMs:  The  average  range  of  detection  (BAFP  2); 
p  =  0.001, 69.6  nm.  when  Blue  work  load  was  high,  and  73.0  nm.  when  work  load  was  low. 
The  average  range  at  which  the  FPBs  were  destroyed  (BSP  4);  p  =  0.001,  27.4  nm.  when 
Blue  work  load  was  high,  and  32.9  nm.  when  Blue  work  load  was  low.  In  both  cases  Blue’s 
performance  was  adversely  affected  (range  decreases)  when  work  load  increased. 

3.  FPB  C^ 

Ho3  had  a  significant  impact  on  the  following  seven  PMs: 

•  Percentage  of  FPBs  detected  by  Blue  Air  (BAFP  1);  p  =  0.000,  90. 1%  detected 
when  FPBs  were  coordinated,  and  98.1%  detected  when  the  FPBs  were  not 
coordinated. 
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•  Mean  range  of  detection  of  the  FPBs  (BAFP  2);  p  =  0.000,  76  nm.  when  FPBs 
were  coordinated,  and  66.6  nm.  when  FPBs  were  not  coordinated. 

•  Percentage  of  FPBs  destroyed  by  Blue  Surface  (BSP3);  p  =  0.000,  65.5% 
destroyed  when  the  FPBs  were  coordinated,  and  84.9%  destroyed  when  the 
FPBs  were  not  coordinated. 

•  Mean  range  at  which  the  FPBs  were  destroyed  by  Blue  Surface  (BSP  4);  p  = 
0.002,  32.8  nm.  when  the  FPBs  were  coordinated,  and  27.5  nm.  when  the  FPBs 
were  not  coordinated. 

•  Number  of  TAO  weapon  assignments  (BAFP  12);  p  =  0.000,  17.0  assignments 
when  the  FPBs  were  coordinated,  and  22.3  assignments  when  the  FPBs  were 
not  coordinated. 

•  Percentage  of  Blue  Air  detected  by  the  FPBs  (A  1);  p  =  0.012,  93.9%  detected 
when  the  FPBs  were  coordinated,  and  97.9%  detected  when  the  FPBs  were  not 
coordinated. 

•  Percentage  of  Blue  Surface  detected  by  the  FPBs  (S  1);  p  =  0.003,  19.3% 
detected  when  the  FPBs  were  coordinated,  and  27.1%  detected  when  the  FPBs 
were  not  coordinated. 

When  the  FPBs  used  high  fewer  FPBs  were  detected  by  Blue,  however,  of  those 
FPBs  detected  the  range  of  detection  by  Blue  was  greater.  Similarly,  when  the  FPBs  had 
high  fewer  were  destroyed,  but  those  that  were  destroyed  were  done  so  at  a  greater  range. 
It  only  makes  sense  that  if  they  were  not  detected  they  could  not  be  killed,  and  if  they  were 
detected  further  out,  that  they  would  be  killed  further  out. 

But  why  would  fewer  be  detected  and  yet,  those  detected,  be  detected  at  greater 
range?  The  answer  to  this  is  inherent  in  the  scenario  design  and  is  two  fold.  When  the 
FPBs  had  high  C^,  they  conducted  a  coordinated  (concurrent)  attack  versus  an 
uncoordinated  (sequential)  attack  with  seven  of  the  nine  FPBs  in  EMCON.  Without  the 
FPBs  radiating  they  were  more  difficult  to  detect,  but  due  to  their  more  structured 
organi2ation  Blue  had  more  time  to  prepare  and  deploy  forces. 

Fewer  TAO  assignments  were  made  when  the  FPBs  used  high  C^.  This  explains 
why  fewer  FPBs  were  destroyed  when  was  high.  Analyzing  the  operations  from  the 
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FPBs  perspective  reveals  that  the  FPBs  detected  fewer  Blue  Air  and  Surface  platforms 
when  they  used  high  C^.  This  is  a  direct  result  and  a  potential  disadvantage  of  the  high  C^, 
EMCON  tactics. 

4.  The  Combined  Effects  of  Information  Completeness  and  FPB  C^ 

Ho5  had  a  significant  impact  on  one  PM:  Mean  range  at  which  the  FPBs  were 
destroyed  by  Blue  Surface  (BSP  4);  p  =  0.011.  When  the  FPBs  were  conducting  non- 
coordinated  attacks,  information  completeness  had  no  effect.  But  when  the  FPBs  used 
coordinated  tactics,  the  FPBs  were  destroyed  at  a  much  greater  range  when  Blue  had 
complete  information,  36.5  nm.,  than  when  Blue  had  incomplete  information,  29.0  nm 
This  indicates  that  when  Red  was  in  EMCON,  good  intelligence  (complete  information) 
became  more  valuable. 

5.  The  Combined  Effects  of  Work  Load  and  FPB  C^ 

Ho6  had  a  significant  impact  on  one  PM  even  though  neither  factor  taken  alone  was 
significant  for  that  PM. 

Mean  range  at  which  Blue  Air  was  first  detected  (A  2):  Additional  analysis  does 
not  yet  help  explain  why  the  FPBs  detected  Blue  Air  further  out  when  conducting 
coordinated  attacks:  p  =  0.000,  46.14  nm.  compared  with  44.56  nm.  when  conducting  non- 
coordinated  attacks. 

6.  The  Combined  Effects  Of  Information  Completeness,  Work  Load,  and 
FPBC^ 

Ho7  had  a  significant  impact  on  one  PM. 

Average  range  of  detection  of  the  FPBs  (BAFP  2):  No  attempt  was  made  in  this 
analysis  to  interpret  this  interaction. 

B.  SUMMARY  OF  MOST  SIGNIFICANT  HYPOTHESES  RESULTS 

•  When  Blue  forces  had  complete  information 

•  FPBs  were  destroyed  at  a  greater  range. 

•  When  Blue  forces  were  under  low  work  load 
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•  FPBs  were  detected  and  destroyed  at  a  greater  range. 

•  When  FPBs  attacked  with  high  C^: 

•  The  number  detected  and  destroyed  decreased,  and 

•  The  range  of  detection  and  destruction  increased. 

C.  REAL  WORLD  MEANING 

The  fact  that  subjects  had  five  helicopters  scouting  in  a  geographically  constrained 
littoral  region  probably  reduced  the  importance  of  judicious  resource  allocation  and  thus 
lessened  the  effect  of  information  completeness.  In  essence,  the  subjects  had  nearly  total 
radar  coverage  provided  by  the  helicopters  regardless  of  where  they  directed  them. 

FPB  levels  caused  changes  in  more  PMs  than  any  other  factor.  Probable  causes 
include: 


•  FPBs  were  in  EMCON  when  was  high  which  greatly  affected  their  ability  to 
see  and  be  seen. 

•  When  FPB  was  high  their  tactics  differed  greatly  than  when  their  was  low. 
In  the  former  case,  the  FPBs  converged  on  Blue  in  three  concentric  waves, 
thereby  massing  forces  for  saturation  attacks.  In  the  latter  case,  the  attack  was 
more  sequential  and  random.  The  net  result  might  be  that  even  though  Blue  was 
able  to  conduct  a  more  methodical  and  deliberate  counterattack  when  the  FPBs 
had  high  they  could  not  detect  as  many  and  therefore  could  not  kill  as  many. 
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VI.  RECOMMENDATIONS 


This  chapter  proposes  ways  in  which  to  improve  and  extend  the  experiment. 

A.  CHANGES  TO  THE  EXPERIMENT 

The  experiment  could  be  improved  by  more  realistically  modeling  information 
completeness  and  work  load.  Reducing  the  number  of  helicopters  would  help  determine  if 
information  completeness  is  modeled  correctly.  Fewer  resources  would  force  the  operator 
to  allocate  them  more  judiciously.  Work  load  could  be  varied  between  high  and  low  based 
on  real  world  maximum  and  minimum  shipping  levels  in  the  scenario’s  geographic  area  to 
ensure  applicability  to  present  day  operations.  The  FPB  threat  is  an  emerging  one  so  it  will 
be  more  difficult  to  find  real  world  data  on  which  to  model  FPB  C^  and  tactics  but  this 
aspect  of  the  study  should  be  pursued. 

Due  to  time  and  human  resources  constraints  a  limited  pilot  study  was  conducted 
(seven  different  subjects,  three  trials  each).  A  larger  pilot  study  would  help  ensure  the 
factors  are  modeled  more  accurately  and  the  PMs  (dependent  variables)  are  capturing  the 
dynamic  range  of  the  performance  curve. 

A  more  homogeneous  group  of  subjects  (preferably  surface  warfare  officers)  would 
help  to  further  reduce  the  impact  of  the  learning  effect  and  ensure  that  all  of  the  subjects 
were  starting  with  the  same  baseline  knowledge  of  naval  tactics. 

Configuring  the  software  to  automatically  store  and  transfer  each  subjects’  trial  data 
would  have  expedited  the  data  collection  process  and  would  have  protected  against 
transposition  errors  in  the  data. 

B.  CONTINUATION  OF  THE  EXPERIMENT 

The  easiest  way  to  extend  this  analysis  is  to  segregate  subjects  by  warfare  specialty, 
service,  or  other  demographic  factor  and  then  analyze  the  data  accordingly.  For  instance, 
this  would  help  determine  if  the  changes  in  the  levels  of  information  were  not  registered  in 
more  PMs  because  of  inaccurate  modeling,  or  a  cancellation  effect  due  to  certain  subjects 
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utilizing  the  scripted  message  information  prudently  and  others  acting  regardless  of 
message  content.  Other  ways  include:  replicating  the  experiment  on  another  simulation 
system  to  show  the  affects  of  B&R  artifacts,  using  different  scenarios  to  neutralize  scenario 
effects,  or  investigating  different  factors  or  different  levels  of  studied  factors. 
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APPENDIX  A.  PM  DATA  COLLECTION  SHEET 


CC41O3  0(PERIMENT#1 

05 -16  FEB  96 

Nomo: 

-V  ^ 

>  ‘  .  C  \ 

Scenario: 

Trial: 

Parameters 

pm 

hIe 

BSiSiR 

2.  Ave  Ran^e  at  which  threats  wore  first  detected 

1  UUJ  nm|  nm 

4.  Ave  Range  at  which  threats  were  destroyed 

UajJ  run 

iB,^e  nm 

|6.  Ave  Range  at  which  cruise  missiles  first  detected  | 

25.T  nm 

HSPSiSS 

jq.55  nm 

mmm\ 

8.  Average  range  at  which  cruise  missile  destroyed 

mmmsR 

wmim 

UcdJ  nm 

%:iStajt3G^ith^dftdr41jlsejt^ 

^nRsi^ 

K^WffWkA 

10.  Real  time  taken  to  set  up  defenses 

lEEBuSS 

■ESKSSI 

1 ,96  min 

tii:;?Redt!Sirj|eifit)th:,b^!nhlt^}d^  . 

ES30O3 

imiM 

1 2.  Number  of  TAO  weapons  assignments 

HSH^I 

sec 

mamm 

14.  Real  ave  time  between  TAO  weajxrn  assignments 

sec 

Kia  u  sec 

Unti  sec 

1 1 5. coi3fiderics®:|i^l  tC^ 

^  iJ-. 

Parameters 

RSP 

RSvtBA 

Results 

RSP 

RSvsBS 

Rssutt$ 

B!SShSB 

iill,  Blde;force  air  pldtt^ife  detect^  ;  :i:  ^ 

imno 

2.  Average  range  at  which  platforms  were  first  detected 

EIBSI^QI 

3.  BluefdK3^'aIr  pldtfOTiifeides^^ 

JS£>W  % 

■9KKi 

4.  Average  range  at  which  platforms  were  destroyed 

Uau  nm 

SifllUefprGjBiPirrTil^lfedetecir^  i  : 

wmMm 

■'•icboo  -ff 

6.  Average  range  at  which  cruise  missiles  were  detected 

mj4  nm 

Erg.fy 

;  S>3S  % 

B.  Average  range  at  which  cruise  missiles  were  destroyed 

■pn 

o  co  nm 

OiCt)  nm 

9.  Targets  (HV^t))  destrpyed  by  surtqce  plcatforrhs  s, , 

IKIS9E2 

H9BSH0 

1 0.  Red  force  surface  platforms  destroyed 

mssm 

% 

B^HTF 

1i .  Average  rangd#which:ria*platfo  destroyed 

HisiyijSii 

haa«isinfii 

HSUEBM 

1 2.  Real  time  from  beginning  to  end  of  scenario 

K^JUcyiii 

IS^HSBS 

KBBxaa 

Figure  1.  PM  Data  Collection  Sheet 
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APPENDIX  B.  EXCEL  WORKSHEET 


Name 

Trial 

I  L  C  BAEPl  BAm 

BSP3 

BSP4  BAFP12 

A1  SI  A2  S2 

A3 

Subl 

1 

1 

2 

1 

100.00 

57.89 

66.67 

20.83 

26 

80  16.67  58.50  16.00 

0 

2 

2 

1 

1 

100.00 

60.67 

66.67 

15.17 

30 

100  50.00  41.80  20.00 

0 

3 

1 

1 

2 

100.00 

79.33 

55.56 

18.80 

16 

100  33.33  47.60  21.00 

0 

4 

1 

2 

2 

77.78 

69.00 

44.44 

18.25 

10 

100  16.67  45.28  24.00 

80 

5 

1 

1 

1 

100.00 

57.33 

77.78 

15.86 

20 

100  33.33  40.40  17.50 

0 

6 

2  2 

1 

88.89 

61.62 

66.67 

15.83 

21 

100  50.00  50.00  16.67 

0 

7 

2 

1 

2 

100.00 

66.78 

44.44 

18.50 

18 

80  50.00  47.25  19.67 

0 

8 

2 

2 

_2_ 

100.00 

59.78 

66.67 

19.67 

19 

80  33.33  44.00  21.00 

0 

Sub  2 

1 

2  2 

2 

100.00 

80.56 

66.67 

38.17 

19 

100  0.00  46.00  N 

20 

2 

1 

2 

2 

77.78 

79.57 

88.89 

25.75 

20 

100  33.33  38.40  17.00100 

3 

2 

1 

1 

100.00 

66.56 

88.89 

32.12 

26 

100  33.33  49.80  19.50 

20 

4 

2 

1 

2 

100.00 

82.78 

88.89 

54.62 

19 

100  0.00  48.40  N 

40 

5 

2 

2 

1 

100.00 

67.33 

88.89 

29.25 

32 

16.67  60.00  14.00 

20 

6 

1 

2 

1 

100.00 

70.56 

100.00 

32.00 

25 

100  33.33  40.00  19.50 

20 

7 

1 

1 

2 

100.00 

89.56 

66.67 

25.67 

38 

100  50.00  49.60  16.33  40 

8 

J_ 

J_ 

100.00 

69.22 

100.00 

34.89 

27 

100  16.67  31.00  14.00 

20 

Sub  3 

1 

2 

2 

2 

88.89 

76.25 

77.78 

40.00 

12 

100  0.00  44.40  N 

0 

2 

2 

1 

1 

100.00 

70.56 

88.89 

37.75 

13 

100  0.00  49.00  N 

0 

3 

1 

2 

2 

66.67 

69.33 

55.56 

31.60 

12 

100  16.67  44.40  24.00 

60 

4 

1 

1 

2 

100.00 

79.67 

66.67 

38.57 

18 

100  0.00  58.20  N 

20 

5 

2 

2 

1 

100.00 

63.67 

88.89 

19.62 

22 

100  33.33  49.00  20.00 

20 

6 

1 

1 

1 

100.00 

62.00 

88.89 

25.62 

26 

100  33.33  38.60  19.50 

0 

7 

2 

1 

2 

88.89 

76.62 

88.89 

31.12 

21 

100  33.33  42.40  20.00 

20 

8 

J_ 

_2, 

2. 

100.00 

60.00 

100.00 

19.44 

28 

100  33.33  49.00  17.50 

20 

Sub  4 

1 

1 

1 

1 

100.00 

64.56 

88.89 

39.12 

16 

100  0.00  41.00  N 

0 

2 

1 

2 

2 

77.78 

67.57 

33.33 

35.67 

7 

100  0.00  37.00  N 

80 

3 

2 

1 

2 

88.89 

81.75 

77.78 

56.29 

9 

100  0.00  50.00  N 

0 

4 

2 

1 

1 

100.00 

72.65 

88.89 

32.38 

15 

100  16.67  30.60  17.00 

20 

5 

1 

2 

1 

100.00 

74.67 

77.78 

26.71 

21 

100  33.33  40.00  17.00 

20 

6 

2 

2 

1 

100.00 

71.11 

88.89 

27.62 

22 

100  33.33  42.60  20.00 

20 

7 

1 

1 

2 

100.00 

94.44 

77.78 

53.29 

10 

100  0.00  50.40  N 

40 

8 

2 

2 

_2. 

100.00 

77.56 

77.78 

46.57 

9 

100  0.00  26.40  N 

0 

Sub  5 

1 

2  2 

1 

100.00 

56.44 

77.78 

22.86 

16 

100  16.67  53.00  17.00 

20 

2 

2 

1 

2 

100.00 

70.56 

77.78 

37.00 

23 

100  0.00  39.00  N 

0 

3 

1 

1 

2 

100.00 

68.33 

66.67 

21.67 

17 

80  16.67  45.00  18.00 

0 

4 

1 

2 

2 

88.89 

63.00 

55.56 

15.60 

22 

100  33.30  40.20  17.00 

80 

5 

2  2 

2 

100.00 

78.11 

55.56 

28.68 

16 

100  16.67  41.00  14.00  40 

6 

1 

2 

1 

88.89 

63.62 

77.78 

28.57 

12 

100  16.67  39.40  14.00 

20 

7 

2 

1 

1 

77.78 

68.43 

77.78 

28.57 

13 

100  16.67  39.80  14.00 

60 

8 

1 

1 

100.00 

71.78 

100.00  46.11 

17 

100  0.00  40.00  N 

20 

Figure  2.  Worksheet,  Subjects  1-5 
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Name 

Trial 

I  L  C  BAFPl  BAFP2 

BSE3 

BSP4  BAEP12 

A1  SI  A2 

S2 

A3 

Sub  6 

1 

2 

2 

1 

100.00 

64.89 

77.78 

26.86 

35 

80  33.33  48.75 

16.50 

20 

2 

1 

1 

2 

44.44 

78.75 

22.22 

18.50 

25 

80  16.67  48.25 

14.00 

0 

3 

2 

1 

2 

44.44 

75.75 

22.22 

30.00 

28 

60  33.33  58.67 

15.50 

0 

4 

2 

1 

1 

88.89 

71.62 

77.78 

26.29 

31 

100  16.67  49.60 

15.00 

0 

5 

2 

2 

2 

55.56 

74.80 

22.72 

19.00 

22 

100  16.67  49.60 

14.00 

0 

6 

1 

1 

1 

88.89 

65.38 

77.78 

26.29 

26 

100  16.67  42.40 

14.00 

20 

7 

1 

2 

2 

77.78 

79.14 

33.33 

24.67 

25 

80  16.67  54.75 

15.00 

0 

8 

1 

2 

1 

100.00 

69.11 

77.78 

22.43 

29 

100  16.67  48.80 

15.00 

0 

Sub  7 

1 

1 

1 

2 

88.89 

76.00 

77.78 

17.57 

18 

100  33.33  45.00 

17.50 

40 

2 

2 

2 

1 

100.00 

69.56 

88.89 

23.38 

15 

100  33.33  41.60 

15.00 

20 

3 

1 

2 

1 

100.00 

67.11 

88.89 

21.50 

19 

100  50.00  47.60 

18.33 

20 

4 

1 

1 

1 

100.00 

67.89 

88.89 

17.62 

30 

100  33.33  41.20 

14.50 

0 

5 

1 

2 

2 

88.89 

73.38 

55.56 

12.60 

18 

100  33.33  30.80 

19.00 

20 

6 

2 

1 

1 

100.00 

69.44 

88.89 

22.28 

29 

100  33.33  39.60 

17.00 

0 

7 

2 

2 

2 

100.00 

78.22 

77.78 

25.00 

18 

80  16.67  44.25 

19.00 

40 

8 

2 

1 

2 

100.00 

81.67 

77.78 

42.00 

17 

60  16.67  59.67 

14.00 

0 

Sub  8 

1 

2 

1 

2 

77.78 

67.14 

33.33 

56.33 

20 

60  0.00  40.67 

N 

0 

2 

2 

2 

1 

100.00 

64.33 

77.78 

13.29 

34 

80  50.00  41.25 

16.33 

0 

3 

1 

1 

1 

100.00 

61.89 

88.89 

24.38 

27 

80  33.33  56.50 

17.50 

0 

4 

1 

2 

1 

100.00 

57.00 

77.78 

17.14 

43 

80  33.33  57.50  21.00 

0 

5 

2 

1 

1 

100.00 

68.89 

100.00 

26.44 

24 

100  33.33  39.80  20.50 

20 

6 

1 

2 

2 

100.00 

72.67 

66.67 

28.50 

19 

100  33.33  45.60 

14.00 

40 

7 

2 

2 

2 

100.00 

67.78 

88.89 

24.88 

19 

100  16.67  46.80 

14.00 

20 

8 

2^ 

100.00 

75.67 

77.78 

34.57 

11 

80  16.67  56.50 

14.00 

20 

Sub  9 

1 

1 

2 

2 

44.44 

70.75 

33.33 

22.67 

10 

80  16.67  46.75  20.00 

60 

2 

1 

1 

1 

100.00 

66.44 

88.89 

39.38 

13 

100  16.67  41.40  20.00 

40 

3 

2 

2 

2 

100.00 

74.56 

66.67 

47.33 

13 

100  0.00  42.20 

N 

40 

4 

2 

2 

1 

100.00 

74.11 

88.89 

42.62 

14 

100  0.00  46.80 

N 

40 

5 

1 

1 

2 

100.00 

90.78 

66.67 

42.00 

17 

100  16.67  50.00  25.00 

20 

‘6 

2 

1 

1 

100.00 

70.33 

100.00 

34.00 

16 

100  50.00  30.40  20.00 

20 

7 

1 

2 

1 

100.00 

72.67 

100.00 

38.89 

11 

80  0.00  47.75 

N 

40 

8 

_2, 

_2, 

100.00 

82.67 

77.78 

45.86 

6 

100  16.67  47.60 

14.00 

20 

Sub  10 

1 

1 

1 

2 

77.78 

75.71 

44.44 

32.25 

21 

100  0.00  46.80 

N 

20 

2 

1 

2 

1 

100.00 

68.78 

77.78 

42.00 

0 

100  16.67  56.00 

15.00 

0 

3 

2 

2 

1 

100.00 

67.56 

88.89 

24.62 

27 

100  50.00  55.60 

19.00 

20 

4 

2 

2 

2 

100.00 

81.11 

77.78 

48.14 

10 

80  16.67  42.75 

15.00 

40 

5 

1 

2 

2 

100.00 

67.44 

66.67 

25.17 

23 

80  33.33  32.25  17.50 

80 

6 

2 

1 

2 

100.00 

82.44 

100.00 

50.22 

18 

80  16.67  45.00 

14.00 

0 

7 

1 

1 

1 

100.00 

71.67 

88.89 

34.88 

27 

100  33.33  33.60  20.00 

20 

8 

2 

1 

1 

100.00 

71.78 

100.00  41.33 

12 

100  0.00  39.60 

N 

0 

Figure  3.  Worksheet,  Subjects  6-10 
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Name 

Trial 

I  L  C  BAEPl  BAEP2 

BSE3 

BSP4  BAFP12 

A1  SI  A2  S2 

A3 

Sub  11 

1 

2 

1 

1 

100.00 

71.67 

66.67 

26.67 

27 

100  16.67  60.00  14.00 

20 

2 

2 

2 

2 

100.00 

77.56 

44.44 

27.50 

19 

100  16.67  46.00  18.00 

0 

3 

1 

2 

1 

100.00 

70.78 

77.78 

28.00 

23 

100  33.33  42.20  16.00 

0 

4 

1 

1 

1 

100.00 

69.00 

100.00 

23-67 

23 

100  16.67  40.60  20.00 

20 

5 

2 

1 

2 

88.89 

83.00 

77.78 

33.00 

13 

100  16.67  55.30  14.00 

0 

6 

1 

1 

2 

88.89 

80.75 

88.89 

30.38 

11 

100  0.00  57.20  N 

20 

7 

2 

2 

1 

100.00 

65.00 

88.89 

22.50 

25 

100  16.67  34.40  20.00 

20 

8 

2_ 

2. 

100.00 

75.56 

66.67 

24.17 

23 

100  33.33  47.00  14.50 

80 

Sub  12 

1 

2 

1 

1 

100.00 

61.11 

77.78 

25.57 

29 

100  33.33  45.40  17.50 

80 

2 

1 

2 

1 

100.00 

73.00 

77.78 

24.29 

21 

100  33.33  47,40  17,00 

40 

3 

2 

2 

2 

100.00 

82.56 

66.67 

39.00 

16 

100  33.33  50.00  15.50 

60 

4 

2 

2 

1 

100.00 

62.67 

88.89 

30.00 

23 

100  16.67  50.20  15.00 

40 

5 

2 

1 

2 

100.00 

81.67 

66.67 

44.33 

23 

100  16.67  48.40  19.00 

20 

6 

1 

2 

2 

88.89 

71.25 

66.67 

29.83 

23 

100  16.67  40.60  15.00 

80 

7 

1 

1 

1 

100.00 

68.33 

88.89 

22.00 

36 

100  50.00  40.00  20.00 

20 

8 

J_ 

2_ 

100.00 

69.78 

100.00 

32.67 

21 

100  33.33  48.20  14.00 

0 

Sub  13 

1 

1 

2 

2 

88.89 

71.00 

66.67 

23.83 

17 

100  33,33  47.40  17.00 

80 

2 

2 

2 

2 

100.00 

78.22 

66.67 

44.50 

15 

100  0.00  53.28  N 

20 

3 

1 

1 

1 

100.00 

68.22 

88.89 

32.25 

18 

100  16.67  33.60  20.00 

0 

4 

1 

2 

1 

100.00 

69.56 

88.89 
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Figure  4.  Worksheet,  Subjects  11-15 
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Figure  5.  Worksheet,  Subject  16 
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Figure  15.  Frequency  Plot  for  A3 


39 


40 


APPENDIX  D.  BOX  PLOTS 


Figure  16.  BAFP  1  Box  Plots 
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Figure  17.  BAFP  2  Box  Plots 
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Figure  18.  BSP  3  Box  Plots 
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BAFP12  BA 


Figure  20.  BAFP  12  Box  Plots 
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Figure  22.  S  1  Box  Plots 
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APPENDIX  E.  INTERACTION  PLOTS 


Figure  26.  BAFP  1  Interaction  Plots 
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Figure  27.  BAFP  2  Interaction  Plots 
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Figure  28.  BSP  3  Interaction  Plots 
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Figure  30.  BAFP 12  Interaction  Plots 
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Interaction  Rot  -  l\/feans  for  A 1 


Figure  31.  A 1  Interaction  Plots 


Interaction  Plot  -  l\/feans  for  S 1 


Figure  32.  S  1  Interaction  Plots 


Figure  33.  A  2  Interaction  Plots 
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Interaction  Plot  -  IVfeans  for  S  2 


Figure  34.  S  2  Interaction  Plots 
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Figure  35.  A3  Interaction  Plots 
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APPENDIX  F. 


Figure  36. 
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Figure  38.  BSP  3  Residuals  Plots 
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Figure  40.  BAFP  12  Residuals  Plots 


Residual 


Figure  41.  A 1  Residuals  Plots 
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Figure  43.  A  2  Residuals  Plots 
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Figure  44.  A3  Residuals  Plots 
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APPENDIX  G.  DEMOGRAPHIC  DATA 


F^re  45.  Q4:  Years  Active  Duty 


Q5:  Proficiency  with  NTD$  Symtootogy 


4  Surface  Warfare  Officers” 
1  Air  Defense  Control 


Frequency 


Figure  46.  Q5:  Proficiency  with  NTDS  Symbology 


71 


Figure  47.  Q6:  Proficiency  with  Wargames 


Figure  48.  Q7:  Proficiency  as  TAO 
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Figure  49.  Q8:  Computer  Proficiency 
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APPENDIX  H.  NULL  HYPOTHESES  REJECTION  SUMMARY 


BAFPl 

BAFP2 

BSP3 

BSP4 

BAFP12 

A1 

SI 

A2 

S2 

A3 

♦ 

II9II 

♦ 

♦ 

■1 

EHHi 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

El 

1  H„, 

mm 

♦ 

1 

n 

IlH 

♦ 

Note:  Reading  this  table  horizontally  provides  a  summary  of  the  PMs  for  which  the  given  null  hypotheses  were  rejected.  Reading  the 
table  vertically  provides  an  indication  of  the  number  of  null  hypotheses  rejected  for  the  given  PM. 


Table  1.  Rejection  Summary 
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APPENDIX  I.  NULL  HYPOTHESES 


Information  Completeness  has  no  impact  on  SAG  performance. 

Ha, 

Information  Completeness  impacts  SAG  performance. 

H«2 

Work  Load  has  no  impact  on  SAG  performance. 

Ha2 

Work  Load  impacts  SAG  performance. 

Ho3 

FPB  C^  has  no  impact  on  SAG  performance. 

Ha3 

FPB  C^  impacts  on  SAG  performance. 

H„4 

The  combined  effect  of  Information  Completeness  and  Work  Load  has  no  impact  on 
SAG  performance. 

H34 

The  combined  effect  of  Information  Completeness  and  Work  Load  impacts  SAG 
performance. 

H„5 

The  combined  effect  of  Information  Completeness  and  FPB  C^  has  no  impact  on 
SAG  performance. 

Has 

The  combined  effect  of  Information  Completeness  and  FPB  C^  impacts  on  SAG 
performance. 

Hos 

The  combined  effect  of  Work  Load  and  FPB  C^  has  no  impact  on  SAG  performance. 

Has 

The  combined  effect  of  Work  Load  and  FPB  C^  impacts  on  SAG  performance. 

Ho7 

The  combined  effect  of  Information  Completeness,  Work  Load  and  FPB  C^  has  no 
impact  on  SAG  performance. 

Ha7 

The  combined  effect  of  Information  Completeness,  Work  Load  and  FPB  C^  impacts 
SAG  performance. 

Table  2.  Null  Hypotheses 
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